Goto

Collaborating Authors

 Golden Gate







Deep networks learn to parse uniform-depth context-free languages from local statistics

arXiv.org Machine Learning

Understanding how the structure of language can be learned from sentences alone is a central question in both cognitive science and machine learning. Studies of the internal representations of Large Language Models (LLMs) support their ability to parse text when predicting the next word, while representing semantic notions independently of surface form. Yet, which data statistics make these feats possible, and how much data is required, remain largely unknown. Probabilistic context-free grammars (PCFGs) provide a tractable testbed for studying these questions. However, prior work has focused either on the post-hoc characterization of the parsing-like algorithms used by trained networks; or on the learnability of PCFGs with fixed syntax, where parsing is unnecessary. Here, we (i) introduce a tunable class of PCFGs in which both the degree of ambiguity and the correlation structure across scales can be controlled; (ii) provide a learning mechanism -- an inference algorithm inspired by the structure of deep convolutional networks -- that links learnability and sample complexity to specific language statistics; and (iii) validate our predictions empirically across deep convolutional and transformer-based architectures. Overall, we propose a unifying framework where correlations at different scales lift local ambiguities, enabling the emergence of hierarchical representations of the data.


Meet the new biologists treating LLMs like aliens

MIT Technology Review

By studying large language models as if they were living things instead of computer programs, scientists are discovering some of their secrets for the first time. How large is a large language model? Think about it this way. In the center of San Francisco there's a hill called Twin Peaks from which you can view nearly the entire city. Picture all of it--every block and intersection, every neighborhood and park, as far as you can see--covered in sheets of paper. Now picture that paper filled with numbers. LLMs contain a LOT of parameters. That's one way to visualize a large language model, or at least a medium-size one: Printed out in 14-point type, a 200-billion-parameter model, such as GPT4o (released by OpenAI in 2024), could fill 46 square miles of paper--roughly enough to cover San Francisco.


More than 20,000 still without power after massive San Francisco blackout

Los Angeles Times

Things to Do in L.A. Tap to enable a layout that focuses on the article. This is read by an automated voice. Please report any issues or inconsistencies here . After Saturday's blackout, roughly 110,000 San Francisco residents have power again. About 21,000 are still in the dark as extensive repairs continue after a substation fire.


A San Francisco power outage left Waymo's self-driving cars stranded at intersections

Engadget

LG TVs add'delete' option for Copilot A San Francisco power outage left Waymo's self-driving cars stranded at intersections Waymo halted its autonomous ride-hailing services in the city in response. Several of Waymo's autonomous vehicles were seen stuck in the middle of San Francisco streets following a significant power outage that took out the city's traffic lights. Waymo responded to the power outage by suspending its ride-hailing services in the city, but images and videos on social media showed the self-driving taxis stopped at intersections with hazard lights on. We have temporarily suspended our ride-hailing services in the San Francisco Bay Area due to the widespread power outage, Suzanne Philion, a spokesperson for Waymo, told Engadget in an email. Our teams are working diligently and in close coordination with city officials, and we are hopeful to bring our services back online soon.